Add Kimi K2.6 NVFP4 B300 EAGLE3 AgentX benchmark / 新增 Kimi K2.6 NVFP4 B300 EAGLE3 AgentX 基准测试#2228
Conversation
cquil11
added
NVIDIA
full-sweep-enabled
agentx
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
cquil11
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Claude finished @cquil11's task in 4m 43s —— View job Review of PR #2228
SummaryOne blocking issue found (see inline comment): the new Everything else checks out: the image is a public Docker Hub reference, |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - config-keys: | ||
| - kimik2.5-fp4-b300-vllm-agentic-mtp | ||
| description: | ||
| - "Add EAGLE3 speculative-decoding arm for the Kimi K2.6 NVFP4 B300 AgentX recipe (draft lightseekorg/kimi-k2.6-eagle3-mla, TOKENSPEED_MLA attention backend with TRT-LLM ragged MLA kernel)." | ||
| - "Add EAGLE3 speculative-decoding arm for the Kimi K2.6 NVFP4 B300 AgentX recipe (draft lightseekorg/kimi-k2.6-eagle3.1-mla, TOKENSPEED_MLA attention backend with TRT-LLM ragged MLA kernel)." | ||
| - "TP8/TP4 GPU-only KV points plus a TP4 native CPU-offload ladder via SimpleCPUOffloadConnector with lazy_offload off; TP4/DCP4 high-concurrency points (conc 32/64) using num_speculative_tokens=3 and synthetic_acceptance_length=2.88." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2158 |
There was a problem hiding this comment.
🔴 BLOCKING: New perf-changelog.yaml entry is inserted in the middle of the file, not appended to the end — and it is character-for-character identical to the entry that already sits at the end of the file (pr-link #2222).
Why it matters: The file is read chronologically (oldest at top, newest at bottom); inserting above existing entries breaks the append-only ordering convention. This happened mechanically because the revert-of-revert re-applied #2158's diff at its original position, while main has since gained the #2222 entry below it.
Fix: Move this entry to the very bottom of the file (below the #2222 entry). Since it re-lands the config, also consider pointing pr-link at this PR (#2228) rather than the original #2158 — that matches what #2222 did — and fixing the stale "(conc 32/64)" in the description, since the DCP4 arms now sweep conc 32–160.
| - config-keys: | |
| - kimik2.5-fp4-b300-vllm-agentic-mtp | |
| description: | |
| - "Add EAGLE3 speculative-decoding arm for the Kimi K2.6 NVFP4 B300 AgentX recipe (draft lightseekorg/kimi-k2.6-eagle3-mla, TOKENSPEED_MLA attention backend with TRT-LLM ragged MLA kernel)." | |
| - "Add EAGLE3 speculative-decoding arm for the Kimi K2.6 NVFP4 B300 AgentX recipe (draft lightseekorg/kimi-k2.6-eagle3.1-mla, TOKENSPEED_MLA attention backend with TRT-LLM ragged MLA kernel)." | |
| - "TP8/TP4 GPU-only KV points plus a TP4 native CPU-offload ladder via SimpleCPUOffloadConnector with lazy_offload off; TP4/DCP4 high-concurrency points (conc 32/64) using num_speculative_tokens=3 and synthetic_acceptance_length=2.88." | |
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2158 |
| echo "Starting vllm server..." | ||
| export PYTHONNOUSERSITE=1 | ||
|
|
||
| export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm | ||
|
|
||
| { set +x; } 2>/dev/null | ||
| VLLM_CMD=( | ||
| vllm serve "$MODEL_PATH" --served-model-name "$MODEL" | ||
| --host 0.0.0.0 | ||
| --port "$PORT" | ||
| --kv-cache-dtype fp8 | ||
| --trust-remote-code | ||
| --block-size 64 | ||
| --language-model-only | ||
| --gpu-memory-utilization 0.90 | ||
| --max-num-seqs "$CONC" | ||
| "${ATTN_BACKEND_ARGS[@]}" | ||
| --attention-config "$ATTN_CONFIG" | ||
| --compilation-config "$COMPILATION_CONFIG" | ||
| --max-cudagraph-capture-size 2048 | ||
| --max-num-batched-tokens 16384 |
There was a problem hiding this comment.
🔴 The new EAGLE3 MTP agentic script (kimik2.5_fp4_b300_mtp.sh) sets --language-model-only but never passes --tool-call-parser kimi_k2 or --reasoning-parser kimi_k2, unlike every other agentic vLLM script for this model family (e.g. kimik2.5_fp4_b300.sh:65-66, kimik2.5_fp4_b200.sh:175-176). Since this is a tool-calling AgentX benchmark, vLLM won't parse tool calls/reasoning content from the model's output during replay, which can break or invalidate the benchmark.
Extended reasoning...
The bug: benchmarks/single_node/agentic/kimik2.5_fp4_b300_mtp.sh builds its VLLM_CMD array (lines 87-107) with --language-model-only but omits --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2. This script serves the model for an agentic-coding (tool-use) AgentX replay via build_replay_cmd / run_agentic_replay_and_write_outputs, which drives /v1/chat/completions requests against the inferencex-agentx-mvp tool-calling trace corpus.
Established pattern this diverges from: every sibling agentic vLLM script for a tool-calling model sets both parser flags. The closest sibling, kimik2.5_fp4_b300.sh:65-66 (same Kimi K2 family, same B300 SKU, same agentic-coding scenario, non-speculative variant of this exact recipe), sets --reasoning-parser kimi_k2 --tool-call-parser kimi_k2. kimik2.5_fp4_b200.sh:175-176 and the int4 Kimi variants do the same. Critically, minimaxm3_fp8_h100.sh:106,109-110 combines --language-model-only WITH --tool-call-parser minimax_m3 --reasoning-parser minimax_m3 (plus --enable-auto-tool-choice) — proving --language-model-only (which disables the vision tower / non-LM submodules) does not replace or preclude the tool-call/reasoning parser flags. The new MTP script appears to be a derivative of kimik2.5_fp4_b300.sh that dropped these two flags during the EAGLE3/DCP rework.
Why nothing else catches this: the script has no validation that parser flags are set when the scenario is agentic-coding; check_env_vars only checks that required env vars exist, not that the constructed VLLM_CMD includes tool/reasoning parsing. The server will start successfully and pass wait_for_server_ready regardless, so the omission is silent until requests carrying tools/tool_choice are replayed.
Impact: without --tool-call-parser kimi_k2, vLLM either (a) rejects tool_choice-bearing chat-completions requests outright, or (b) accepts them but never structures the model's tool-call text into tool_calls, degrading the trace replay into plain text completion. Since benchmark_lib.sh enforces a 10% failed-request threshold for the sweep, case (a) would push a tool-calling-heavy trace over that threshold and fail the run; case (b) would silently produce throughput numbers for a benchmark that no longer measures actual tool-calling behavior, defeating the purpose of the AgentX agentic-coding scenario this PR is adding.
Proof walkthrough:
configs/nvidia-master.yamlregisterskimik2.5-fp4-b300-vllm-agentic-mtpunder theagentic-codingscenario.- The sweep runner invokes
kimik2.5_fp4_b300_mtp.sh, which launchesvllm servewith theVLLM_CMDarray shown in the diff — no--tool-call-parser/--reasoning-parserpresent, only--language-model-only. resolve_trace_source/build_replay_cmdconfigure aiperf to replay theinferencex-agentx-mvpcorpus, which contains tool-definition-bearing chat-completions requests (this is what makes the scenario "agentic-coding" rather than plain chat).- When aiperf sends a request with
tools/tool_choiceto a vLLM server started without a tool-call parser, vLLM either 400s the request or returns unparsed tool-call text as plain content, instead of populatingtool_calls. - Compare with
kimik2.5_fp4_b300.sh(same model family, same replay path) which explicitly sets--reasoning-parser kimi_k2 --tool-call-parser kimi_k2at lines 65-66 — confirming this is the required, intentional configuration for this exact benchmark type, not an optional flag that only some scripts happen to add.
Fix: add --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2 to the VLLM_CMD array in kimik2.5_fp4_b300_mtp.sh, matching kimik2.5_fp4_b300.sh:65-66.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29455077426 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29455077426 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29455077426 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29471460196 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29455077426 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29455077426 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29471934256 |
4 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29471934256 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29471934256 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29471934256 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29471934256 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29506569772 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29506569772 |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
# Conflicts: # perf-changelog.yaml
5 participants
Adds an EAGLE3 speculative-decoding AgentX (agentic-coding) benchmark for Kimi K2.6 NVFP4 on B300, served with upstream vLLM (
vllm/vllm-openai:nightly-94c0ef30…).Changes
kimik2.5-fp4-b300-vllm-agentic-mtp(configs/nvidia-master.yaml), runnercluster:b300-nv. Search space: TP8, TP4, TP4 + native CPU KV offload (SimpleCPUOffloadConnector,dram-utilization: 0.80), and TP4/DCP4 with and without offload at high concurrency.benchmarks/single_node/agentic/kimik2.5_fp4_b300_mtp.sh: EAGLE3 draftlightseekorg/kimi-k2.6-eagle3.1-mlawith simulated synthetic acceptance — non-DCP arms usenum_speculative_tokens=4/ AL 3.24; DCP4 arms usenum_speculative_tokens=3/ AL 2.88.TOKENSPEED_MLAattention backend,TRTLLM_RAGGEDMLA prefill (non-DCP), fp8 KV cache.Kimi-K2.6-NVFP4to the b300-nv model allowlist.process_changelog.pynow emits JSON withexclude_none=True; perf-changelog entry added.Validation
/stage-results)为 B300 上的 Kimi K2.6 NVFP4 新增 EAGLE3 投机解码 AgentX（agentic-coding）基准测试，使用上游 vLLM 镜像。新增
kimik2.5-fp4-b300-vllm-agentic-mtp配置（TP8 / TP4 / TP4+CPU KV offload / TP4+DCP4 组合）、基准脚本（EAGLE3 草稿模型lightseekorg/kimi-k2.6-eagle3.1-mla，非 DCP 使用 spec4/AL 3.24，DCP4 使用 spec3/AL 2.88）、b300-nv 模型白名单及 perf-changelog 条目。Original PR: #2158